Practical Uncertainty Minimization for Spectral Macrostate Data Clustering

Spectral clustering, which uses the global information embedded in eigenvectors of an inter-item relation matrix, can outperform traditional approaches such as k-means and hierarchical clustering. Spectral hierarchical bipartitioning is well-understood, but spectral multipartitioning remains an interesting research topic. Korenblum and Shalloway [Phys. Rev. E 67, 056704 (2003)] used an analogy to the dynamic coarse-graining of a stochastic system and the principle of cluster uncertainty minimization to motivate a fuzzy spectral multipartitioning method, macrostate data clustering (MDC), that could solve problems that defeated other methods. However, MDC poses a challenging non-convex global optimization problem that was solved by a brute-force technique unlikely to scale to problem sizes beyond O(10). Here we provide further tests of the accuracy of MDC and develop a new method for solving the optimization problem, which scales to data sets at least two orders-ofmagnitude larger. This range includes problems of significant biological interest, such as microarray analysis of gene expression data. Moreover, we show that the method of Weber et al. [Tech. Rep. 04-39, Konrad-Zuse-Zentrum für Informationstechnik Berlin (2004)] provides a zeroth-order solution to the minimum uncertainty problem and provide a new geometric interpretation of the solution. We show how this approximation can be naturally extended to an exact solution and how the conditions for its validity can be extended past those previously proposed.